Methods, Devices, and Systems for Non-Mechanically Restricting and/or Programming Movement of a Tool of a Manipulator Along a Single Axis

ABSTRACT

Methods, devices (such as computer readable media), and systems (such as computer systems) for performing movements of a tool of a medical robot along a single axis that are achieved by electronically limiting the medical robot&#39;s movement to produce movement of the tool along the single axis rather than mechanically restricting the medical robot&#39;s movement to produce the single axis movement. The tool&#39;s movement will be along the single axis even if a user is moving an input device linked to the medical robot in other axes during the single axis movement. In addition, techniques are disclosed for automating the single axis movement such that it can be programmed to stop at a target location and start at or near a second (e.g., starting) location, which is useful for a procedure such as a brain biopsy, breast biopsy or implantation, and such that a user can execute a command instructing the medical robot to perform the movement without the need for the user to manipulate an input device to cause real-time responsive movement of the medical robot.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 60/912,146, filed Apr. 16, 2007, which is incorporated byreference. Co-pending International Application No. PCT/US08/60541 isalso incorporated by reference.

BACKGROUND INFORMATION

The present methods, devices, and systems relate generally to the fieldof surgical robotics, and more particularly to the non-mechanicalrestriction of a manipulator (e.g., a robotic arm with multiple degreesof freedom) to movement of a tool by the manipulator along a singleaxis. An example of a procedure that can be carried out according to thepresent methods, devices, and systems is an automated biopsy. An exampleof a surgical robot that can be used in a procedure to which the presentmethods, devices, and systems relate is disclosed in U.S. Pat. No.7,155,316 (the “'316 patent”), which is incorporated by reference.

In order to perform stereotactic procedures (e.g., take a needle orsmall tool and hit a target within a three dimensional space) it isadvantageous to limit the extent to which the tool can deviate from itsplanned trajectory. Therefore, in order to use a robot to performstereotactic procedures using a master-slave interface, it can bedesirable to nullify any inputs to the master controllers in the X and Ycoordinates thus restricting movement at the tool tip to the Z axis.

Current procedures using frame-based or frameless stereotactic toolscreate Z-lock conditions through mechanical limitations. The most commonprocess for stereotactic procedures (frame-based) requires the fixtureof a rigid head frame to the patient's head. This frame serves as amechanical means of guiding stereotactic tools through pre-planned pathsby mechanically limiting X and Y axis movement. Other framelessstereotactic tools that use mechanical arms or tool attachments executestereotactic procedures by fixing the patient's head in space,positioning the mechanical arm in a pre-planned path position, andmechanically locking the degrees of freedom associated with the arm. Theresult is a mechanical Z-lock along a pre-planned path.

In both the frame-based and frameless stereotactic procedures, thepre-planned path is derived from an image taken hours before theprocedure. However, the brain is not fixed within the cranial cavity andcan shift as a result of damage, tumours, hydration, and body positionchanges. These relatively small brain shifts can be problematic in termof accuracy and pose a safety concern. As a result, post surgical imagesand other tools are used to ensure accurate and safe procedures withexisting tools. Furthermore, in frame-based stereotactic procedures,attachment of a head frame to the patient's head is also required; thisis both uncomfortable and time consuming.

Significant time is associated with pre-operative planning andpost-surgical imaging. Moreover, frameless stereotaxy navigation systemsrequire line of sight with the patient's head and the surgeon's tools.This can pose a problem for surgeons who need to be positioned by thehead of the patient to navigate stereotactic tools to the target.

SUMMARY

Embodiments of the present methods and systems enable a user, such as asurgeon, to set up and execute an automated move of a tool of one of therobotic arms (which includes a tool that is coupled to the robotic arm,as well as a tool that is integrated with the robotic arm) along asingle axis, such as the longitudinal axis of the tool. Such a move maybe particularly advantageous when implemented as an automated biopsy oftissue, such as brain or breast tissue. The automated move may beprogrammed to occur during a stereotactic procedure, when some or all ofthe robotic arm is positioned within the bore of an open or closedmagnet of a magnetic resonance imaging machine, or during amicrosurgical procedure during which one or both robotic arms may be setup to and execute such an automated move. Robots that may be manipulatedaccording to the present techniques may be characterized ascomputer-assisted devices.

In some embodiments, the present systems take the form of a computersystem useful in simulating, planning and/or executing an automatedsurgical procedure. The computer system is configured to perform atleast the following functions: receive data designating a targetlocation for a tool held by a medical robot; receive data designating asecond location for the tool from which the tool will move toward thetarget location during an automated movement; and move the medical robotin response to a user command to begin the automated movement such thatthe tool moves along a single axis defined by the second location andthe target location. The data designating the target location maycomprise coordinates (e.g., Cartesian coordinates) of the tip of thetool, or coordinates of a location spaced away from the tool along alongitudinal axis of the tool, in any suitable coordinate system, ordata sufficient to enable determination of such coordinates (such asjoint values of the robotic arm that allow forward kinematics to be usedto solve for the coordinates based on known parameters such as roboticarm link lengths).

In some embodiments, the present devices take the form of a computerreadable medium comprising machine readable instructions for receiving acommand to restrict movement of an instrument held by or integral with arobotic arm along a single axis, the robotic arm being configured foruse in surgery; receiving a position and orientation of an input device,the input device being linked to the robotic arm through a master-slaverelationship in which the input device is the master, the differencebetween the position and orientation of the input device and a previousposition and orientation of the input device corresponding to a desiredmovement of the instrument; and sending a signal or signals to effect amove of the instrument in accordance with the desired movement, wherethe move will be along the single axis and will not include any movementalong any different axis from the single axis. The signal or signals maybe any suitable form of data that includes information sufficient tocause the robotic arm to move appropriately. For example, the signal orsignals could represent a set of joint displacements and/or jointvelocities outputted to a local controller for the robotic arm ordirectly to the individual joint actuators.

In some embodiments, the user may set up a procedure by deliveringinputs to a computer system through an input device, such as a handcontroller that is linked as a master to the robotic arm in amaster-slave relationship. The user may also deliver inputs through oneor more graphical user interfaces (GUIs) using any suitable inputdevice, such as touch screen controls (e.g., buttons, slider bars, dropdown menus, tabs, etc.), a mouse, or the like. Some embodiments of thepresent systems are computer systems that may be configured to displayon a display screen a GUI that allows the user to select a simulationmode (e.g., through a control such as a button that can be selected viaa touch, a mouse, or the like) for setting up the automated movement andotherwise for training. The computer system also may be configured todisplay on the GUI one or more controls (e.g., that can be selected viaa touch, a mouse, or the like) for selecting the type of surgery, suchas microsurgery, stereotaxy with one of the robotic aims, or stereotaxywith the other robotic arm. The computer system also may be configuredto display on the GUI one or more controls (e.g., that can be selectedvia a touch, a mouse, or the like) for activating power to: the roboticarms (e.g., through separate buttons); a base motor for adjusting theheight of the base on which the robotic arms sit during microsurgicalprocedures; a digitizing arm usable during the physical registrationprocess for registering a structure (e.g., of a radio-frequency coilassembly) associate in a fixed relationship with a portion of a subjectto one or both robotic arms; a field camera usable during microsurgeryto capture images of the surgical field; and a bore camera or cameras tobe positioned in the bore of a magnet of a magnetic resonance imagingmachine. The computer system also may be configured to display on theGUI one or more controls (e.g., that can be selected via a touch, amouse, or the like) for activating a single axis lock (e.g., a Z-axislock) and another button or buttons for controlling which robotic arm toassociate the single axis lock with.

The computer system also may be configured to display on one or moreadditional display screens one or more additional GUIs for displayingtwo-dimensional images (one at a time) of a portion of a subject and fordisplaying a three-dimensional representation (e.g., a set of 2D imagesthat form a 3D dataset of images representing a volume) of a portion ofa subject. When only one such GUI is provided on one additional displayscreen, the computer system may be configured to display one or morecontrols (e.g., buttons, tabs, or the like that can be selected via atouch, a mouse, or the like) that a user can select to display either 2Dimages (one at a time) or a 3D image. The computer system also may beconfigured to display a zoom button, slider bar, or the like (e.g., thatcan be selected/manipulated via a touch, a mouse, or the like) that willallow a user that has selected the 2D display to zoom in on a given 2Dimage, where the 2D image remains centered as it is enlarged or reducedin size. The computer system also may be configured to display controls(e.g., that can be selected via a touch, a mouse, or the like) thatallow a user to turn on a tracking feature for one of the two roboticarms that will be displayed as crosshairs representative of the locationof either (a) the working tip (e.g., the distal tip) of a tool of therobotic arm selected or (b) the end of a line that extends from the tooltip, and further may be configured to display controls (e.g., buttons,slider bars, or the like that can be selected via a touch, a mouse, orthe like) that allow a user to activate the display of the extensionline and control the length of the extension line. As a user manipulatesan input device linked to a selected robotic arm, and the user'smovement alters (in simulation mode, in which the robotic arm does notactually move) the position of the tool held by/integrated with therobotic arm, the crosshairs move as a result, and the displayed 2D image(if in 2D display mode) changes to match the would-be depth of the tool(or extension line) relative to the subject. Alternatively, a given 2Dimage may comprise an oblique slice that is oriented perpendicular tothe tool axis. Such slices interpolate pixels between the 2D slices toachieve off-axis images. Likewise, the 3D image also changes in responseby 2D slices that make up the 3D image being taken away or addeddepending on the depth of the tool/extension line into the subject.

The computer system may also be configured to display, when either the2D or 3D display is selected, a section corresponding to planning for anautomated biopsy that includes a display of controls (e.g., that can beselected via a touch, a mouse, or the like) that can be used to set atarget location (associated with a location of the crosshairs at thetime when the target location button is selected) for an automatedmovement along a single axis (e.g., an automated biopsy); a second point(characterizable as a start point, though a given movement may not beginexactly at the start point; the second point being associated with alocation of the crosshairs at the time when the second button isselected) that together with the target location defines a path for thetool movement; a tool alignment function, that can be used when a userdesires to position the relevant robotic arm in place (e.g., within apreset distance, ranging from zero to some relatively small distance(e.g., 2 centimeters)) for the automated move procedure, and that whenpressed will move the robotic arm so that the tool tip is positioned onor near the start point; and an execute function that a user can pressin order to start the automated move of the tool, provided that the userenables the input device (e.g., by holding the input device and pushinga button on the input device with the user's finger). The computersystem may also be configured to display an indicator (e.g., a coloredcircle) for the target location selected by the user; a line extendingfrom the indicator and to the tool tip or extension line tip (whicheveris used) following selection of the target location, the line beingdesigned to show the user the path through the subject if the line isfollowed, the computer system being configured to alter the appearanceof the line when a second point is selected (e.g., changing the line'scolor or shape).

Thus, in some embodiments, the computer system may be configured toperform at least the following functions: receive a command (e.g.,through a user's touch of the screen displaying the relevant GUI)identifying a target location for a tool used in an automated movementby a robotic arm; receive a command identifying a starting location forthe tool; receive a command to execute an automated move along a path(e.g., a line) defined at least in part by the starting location and thetarget location; and execute the automated move such that the tool,which may have a longitudinal axis, travels along the path (e.g., alonga single axis). That path also may be aligned with the toolslongitudinal axis. In some embodiments, the computer system may also beconfigured to receive (e.g., prior to the command identifying the targetlocation) a command selecting which robotic arm to use for the automatedmove. In some embodiments, the computer system may also be configured toreceive (e.g., prior to the command identifying the target location) acommand indicating a simulation and/or setup mode that disengages aninput device that is linked in a master-slave relationship to a roboticarm holding or integrated with the tool, such that in the simulationmode movement of the input device does not cause movement of the roboticarm. In some embodiments, the computer system may also be configured toreceive a command (e.g., prior to the command identifying the targetlocation) indicating a user's activation of the input device (such asthrough the user touching a button on in the input device with theuser's hand), which activation allows the user to alter the position ofthe tracking indicator showing the location of the would-be tool tiprelative to the image(s) of the subject as the user determines where toposition the tracking indicator for selection of the target and startinglocations. In some embodiments, the computer system may also beconfigured to receive a command (e.g., after the command identifying thestarting location) indicating a new (e.g., a second) target location. Insome embodiments, the computer system may also be configured to receivea command (e.g., after the command identifying the starting location)indicating a new (e.g., a second) starting location. In someembodiments, the computer system may also be configured to receive acommand (e.g., after the command identifying the starting location)indicating termination of the simulation and/or setup mode. In someembodiments, the computer system may be configured to display on a GUI acontrol (e.g., that can be selected via a touch, a mouse, or the like)that can be used to select a mode in which the input device is engagedwith the robotic arm in a master-slave relationship. In someembodiments, the computer system may be configured to receive a command,when in the master-slave mode, enabling the input device (e.g., byholding the input device and pushing a button on the input device with afinger of the user). In some embodiments, the computer system may beconfigured to receive a command to execute the automated move along apath that is defined at least in part by the starting and targetlocations, the computer system also be configured to cause the roboticarm in a way that moves the tool in a single axis along the path onlyafter it has received a command indicating the input device is enabled(e.g., such that a user must be holding the input device in order forthe automated move to proceed). The computer system may be configured tostop the robotic arm from completing the automated move if it receives acommand to stop the automated move (e.g., through a user pushing thesame button on the input device that otherwise enables the inputdevice), and may also be configured to display on a GUI a message thatincludes buttons or the like (e.g., that can be selected via a touch, amouse, or the like) for continuing with the automated move, reversingdirection, or stopping, and may be configured to receive a command toeither continue, reverse direction or stop, depending on the button orthe like that is activated, provided it first receives a commandindicating the input device is enabled (e.g., by holding the inputdevice and pushing a button on the input device with the user's finger).

Any embodiment of any of the present methods, devices, and systems mayconsist of or consist essentially of—rather thancomprise/include/contain/have—the described steps and/or features. Thus,in any of the claims, the term “consisting of” or “consistingessentially of” may be substituted for any of the open-ended linkingverbs recited above, in order to change the scope of a given claim fromwhat it would otherwise be using the open-ended linking verb.

BRIEF DESCRIPTION OF THE FIGURES

The following drawings illustrate by way of example and not limitation.Identical reference numerals do not necessarily indicate an identicalstructure, system, or display. Rather, the same reference numeral may beused to indicate a similar feature or a feature with similarfunctionality. Every feature of each embodiment is not always labeled inevery figure in which that embodiment appears, in order to keep thefigures clear. The hand controllers, manipulators and tools shown in thefigures are drawn to scale, meaning the sizes of the depicted elementsare accurate relative to each other.

FIG. 1A is a perspective view of one embodiment of two input devices(hand controllers) that may be used consistent with the presenttechniques.

FIG. 1B is an enlarged view of a left-handed input device.

FIGS. 1C-1E are different views showing a tool held by a robotic armlocated in a first position of a stereotactic procedure.

FIGS. 2A-2C are different views showing the tool from FIGS. 1C-1E in asecond position of a stereotactic procedure.

FIGS. 3A-3C are different views showing the tool from FIGS. 1C-1E in athird position of a stereotactic procedure.

FIG. 4 is a perspective view of a workstation for use in planning andcontrolling the tool movement shown in FIGS. 1C-3C.

FIG. 5 shows a graphical user interface that can be used in the set upand control of the tool movement shown in FIGS. 1C-3C.

FIG. 6 shows the GUI of FIG. 5 in a training/simulation mode involvingboth robotic arms.

FIG. 7 shows the GUI of FIG. 5 in a training/simulation mode in whichthe left arm has been chosen for stereotaxy.

FIG. 8 shows the GUI of FIG. 5 in a training/simulation mode in whichthe right arm has been chosen for stereotaxy and the Z Axis Lockfunction for the tool of that arm has been activated.

FIG. 9 shows the GUI of FIG. 5 in a training/simulation mode in whichthe microsurgery application has been chosen, both arms are enabled, andboth tools have been chosen.

FIG. 10 shows the GUI of FIG. 5 in a training/simulation mode in whichthe left arm has been chosen for stereotaxy (as in FIG. 7) and the ZAxis Lock function for the tool of that arm has been activated.

FIG. 11 shows another GUI that can be used in the set up of the toolmovement shown in FIGS. 1C-3C. An indicator in the form of crosshairscorresponding to the location of the tool tip is shown on a 2D image,representing the tool tip's location relative to the portion of thesubject in the 2D image.

FIG. 12 shows the GUI of FIG. 11 in a 3D mode in which a representationof the tool chosen for use in the training/simulation is displayed atthe relevant depth within a 3D image of a portion of the subject. TheGUI reflects that a user has selected the “Plane Cut” option, whichresults in oblique slices being cut away on the head to the relevanttool tip depth.

FIG. 13 shows a warning box that can appear on a GUI such as the one inFIG. 5 if a problem is encountered during an automated procedure.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The terms “comprise” (and any form of comprise, such as “comprises” and“comprising”), “have” (and any form of have, such as “has” and“having”), “contain” (and any form of contain, such as “contains” and“containing”), and “include” (and any form of include, such as“includes” and “including”) are open-ended linking verbs. As a result, amethod, device, or system that “comprises,” “has,” “contains,” or“includes” one or more recited steps or elements possesses those recitedsteps or elements, but is not limited to possessing only those steps orelements; it may possess (i.e., cover) elements or steps that are notrecited. Likewise, an element of a method, device, or system that“comprises,” “has,” “contains,” or “includes” one or more recitedfeatures possesses those features, but is not limited to possessing onlythose features; it may possess features that are not recited. Similarly,a computer readable medium “comprising” (or “encoded with”) machinereadable instructions for performing certain steps is a computerreadable medium that has machine readable instructions for implementingat least the recited steps, but also covers media having machinereadable instructions for implementing additional, unrecited steps.Further, a computer system that is configured to perform at leastcertain functions is not limited to performing only the recitedfunctions, and may be configured in a way or ways that are not specifiedprovided the system is configured to perform the recited functions.

The terms “a” and “an” are defined as one or more than one unless thisdisclosure explicitly requires otherwise. The term “another” is definedas at least a second or more. The terms “substantially” is defined as atleast close to (and includes) a given value or state (preferably within10% of, more preferably within 1% of, and most preferably within 0.1%of).

In some embodiments, the invention is a software enabled single-axislock for movement of a tool along the single axis by a robotic arm withmultiple degrees of freedom. The software solution allows a robotic armwith an unlimited number of degrees of freedom to behave in the samefashion as a robot or device that is mechanically restricted to motionof its tool along the single axis. Prior to a procedure, a command maybe sent to the software to lock the motion by a given robotic arm of itstool (meaning a tool the robotic arm is holding or that is integral withthe robotic arm; the present tools may be characterized morespecifically as medical tools or surgical tools) in a single axis usingany suitable input device, such as a button on a touch screen on a GUI,a button on an input device (e.g., a hand controller), or the like.

The apparatus to which the inventive techniques may be applied may, insome embodiments, include a slave robotic arm commanded by a masterinput device, such as a hand controller. An example of a pair of inputdevices (in the form of hand controllers) that can be used to controltwo different robotic arms, respectively, of a medical or surgicalrobotic system are shown in FIG. 1A. Input devices 20, which are mirrorimages of each other, each includes a stylus 25 that can be held like along pen, lever 27 that can be squeezed toward stylus 25 to cause a toolintegrated with or held by the slave robotic arm to actuate (e.g.,squeezing lever 27 can cause forceps to close), and an enable/disablebutton 29 that can be touched and held for a short amount of time inorder to activate the input device. One way to hold input devices 20 isto grasp stylus 25 so that lever 27 can be squeezed with the forefingerand so that button 29 can be touched with the thumb. FIG. 1B shows anenlarged view of the left-handed input device 20.

Closed or open form forward and inverse kinematic solutions may becreated such that an individual with ordinary skill in the art can usethe joint values characterizing the position of each joint of therobotic arm to solve for a commanded tool tip position (taking intoconsideration the permitted axis of movement), and then take thatcommanded tool tip position and solve for the joint angles that must beachieved to move the surgical tool to the commanded tool tip positionalong a single axis.

One manner of creating a non-mechanical single-axis tool movement lock(after, for example, a command has been received to create one) involvesthe following:

a) retrieving an input device (e.g., hand controller) command in tooltip space (e.g., Cartesian X, Y, Z, roll, pitch, yaw). This retrievingmay comprise receiving a hand controller signal(s) (command(s), or data)signifying the position and orientation of the hand controller;determining (e.g., calculating) a delta value of the movement of thehand controller in a single axis (e.g., an axis that is related by atransformation to the single axis to which tool tip movement isrestricted); and determining a corresponding delta value for the tooltip using that hand controller delta. If a transformation from deltavalues in hand controller space to delta values in tool tip space isdetermined, all tool tip delta values may be ignored except the deltaalong the relevant single axis. This could effectively be achieved byeither a simple zeroing of non single axis parameters received from thehand controller or calculating all the delta values for each axis andusing only the delta value in the single axis direction.

b) take the current position of the manipulator (which is a term thatcan describe the robotic arm) and perform a forward kinematic solutionto get tool tip X, Y, Z, roll, pitch, yaw.

c) add the single axis delta determined in step a) to the currentmanipulator tip position determined in step b).

d) using this new tip position, perform an inverse kinematics to solvefor the required joint angles.

e) command the manipulator to the new joint values.

f) repeat from step a).

In some embodiments, the rate of execution of the above loop may bearbitrarily small to produce linear motion at the tool tip. The longerthe time or the bigger the steps taken, the more non-linearity can becreated as the motion between each Cartesian position is in joint space,and joints are interpreted linearly over the desired range of travel.Smaller motions on the order of 10 milliseconds result in imperceptiblenon-linearities between each Cartesian tip command and an effectivelinear motion.

Furthermore, as discussed in more detail below, in some embodiments themovement of a tool along a single axis may be pre-programmed so as to beautomated.

Referring now to FIGS. 1C-3C, detailed views of a manipulator 100 and asurgical tool 150 are shown in various positions as the manipulator 100causes movement of the tool along a single axis in a stereotacticprocedure (such a movement also may be achieved in any other procedure,such as a microsurgical procedure). Manipulator 100, which is an exampleof a multi-degree of freedom robotic arm (specifically, manipulator 100may be characterized as a six degree of freedom slave manipulator, andit is similar in functionality and operation to the robotic armsdisclosed in the '316 patent), assembly 200 comprising a head clamp andradio-frequency coil device (which is coupled to the head clamp, andwhich can be further coupled to the operating room table by a fixablemulti link arm), and cameras 190 (only one of which is visible (theother is on the opposite side of the extension board)) are coupled to anextension board 260. Extension board 260 may be coupled to any suitabletable or other structure having a patient support surface (not shown).In the views shown, a schematic drawing of a patient's head 300 is shownheld by the head clamp of assembly 200.

FIGS. 1C and 1D show manipulator 100 in a first position in whichsurgical tool 150 is located outside of head 300, near opening 350 inhead 300, which may be a burr hole or any other suitable surgicalopening. However, tip 160 of surgical tool 150 is outside of opening350. FIG. 1E is a side view of the position shown in FIGS. 1C and 1D,and does not include assembly 200 for clarity. FIGS. 2A and 2B showmanipulator 100 moved to a second position in which tip 160 of surgicaltool 150 has been advanced along axis 110, and no other axis, bymanipulator 100 so that tip 160 has penetrated the boundary of opening350. FIG. 2C is a side view of the position shown in FIGS. 2A and 2B,and does not include assembly 200 for clarity. FIGS. 3A and 3B showmanipulator 100 moved to a third position in which tip 160 has movedalong axis 110 into a location within head 300, which it can be furthermanipulated by a user/operator (e.g., a surgeon) to perform any of anumber of functions, such as taking a biopsy of tissue. FIG. 3C is aside view of the position shown in FIGS. 3A and 3B, and does not includeassembly 200 for clarity. Axis 110 is substantially aligned (perfectlyaligned in the depicted embodiment) with the longitudinal axis (notseparately shown) of tool 150. For other tools that have bends orangles, the tool and tool tip will still move along a single axis,however that axis may not coincide with a longitudinal axis of the toolitself.

FIG. 4 illustrates a perspective view of a workstation 400 that can beused to control manipulator 100 (or two such manipulators) and surgicaltool 150 (or two such surgical tools, one held by each of two suchmanipulators). In certain embodiments, workstation 400 comprises inputdevices 20 shown in FIGS. 1A and 1B to control movement of manipulator100. Workstation 400 may include a table to which the input devices aresecured as well as a series of display screens, including displayscreens 401 and 402, each of which can provide a graphical userinterface (GUI) that can be used in setting up a procedure usingmanipulator 100. In a preferred embodiment, the GUI shown on displayscreen 401 may be used to select two points that will define the axis(or path or trajectory) along which the tip of the relevant tool travelsin an automated single axis movement (such a screen is referred to as acommand status display (CSD) in this document) and the GUI shown ondisplay screen 402 may be used to display one or more images from athree-dimensional dataset of images of a subject taken using athree-dimensional imaging device, such as a magnetic resonance imagingdevice, which may be viewed as a determination is made by an operatorabout which points to select on display screen 402 (such a screen isreferred to in this document as a magnetic resonance image display(MRID)). The other display screens depicted in FIG. 4 may be used toshow other images or displays associated with a given procedureinvolving one or both manipulators 100.

FIGS. 5-11 illustrate various screen displays that can be used as GUIsfor displays 401 and 402. As shown in the figures, multiple controls(such as buttons, slider bars, radio buttons, check boxes, drop downmenus, and the like) are provided on each screen for receiving userinput through any suitable means, such as through touching the screen,manipulating an input device such as a mouse, or the like. Only thosecontrols relevant to the features presented in this disclosure will bediscussed.

In certain embodiments, a computer system may be configured such thatstarting the primary application supported by the computer system bringsthe user to a startup screen as illustrated in FIG. 5. Those of ordinaryskill in the art having the benefit of this disclosure will be able towrite code (machine readable instructions, which can be implementedthrough software, hardware, firmware, or a combination of any two ormore of these) without undue experimentation for accomplishing thefeatures (including the graphical user interfaces) described below andshown in the figures. FIG. 5 illustrates a CSD 401 that can be used insetting up a desired mode for one or both manipulators (such as a “ZAxis Lock” representative of a manipulators ability to move its toolalong only one axis) or a desired procedure, such as an automated move(e.g., along a single axis) of a surgical tool by a given manipulator100. This display includes options for selecting procedure types(microsurgery, stereotaxy left arm, stereotaxy right arm), as well aspower selections for the right arm, left arm, a base motor for adjustingthe height of the arms (manipulators 100) supported on a mobile andlockable base, a field camera for capturing images during microsurgeryand a digitizing arm for use in the physical part of subjectimage-to-manipulator registration. The power buttons are shown in the“Mode Controls” tab, as is the “Surgery Type.” Manipulators 100 areshown in an unhighlighted manner on the GUI shown in FIG. 5, signifyingthat neither has been selected for using in either training/simulationor a procedure using the buttons at the bottom left of the screen.

A suitable technique for registering one or more two-dimensional imagesof a portion of a subject with one or both manipulators 100 is disclosedin co-pending International Application No. PCT/US08/60538, which isincorporated by reference. Once suitable registration has beenaccomplished, which may include both an MRI registration aspect tolocate the imaged subject to a physical structure and a physicalregistration aspect to register a given manipulator to that physicalstructure, set up may begin.

In one exemplary embodiment, a user may select a simulation mode byselecting the “Training Simulation Mode” button under the “UserSettings” tab shown in CSD 401 of FIG. 6. Selecting the simulation modecan allow the user to view simulated movements of manipulator 100 andsurgical tool 150 in response to movements of the input device, withoutcausing actual movement of manipulator 100. The word “simulation” alsoappears near the bottom portion of the display, as shown for example inFIGS. 5-7. In simulation mode, a user can view a potential path oftravel of surgical tool 150 that may be used in a surgical procedure.This allows a user to evaluate multiple potential paths of manipulator100 and surgical tool 150 before defining one as described below foractual use in the procedure.

CSD 401 in FIG. 7 illustrates the system in simulated stereotaxy modewith the left arm enabled. This version of CSD 401 now shows only onemanipulator as a result of the left arm selection, and shows it in ahighlighted state. It also shows an upper portion of an RF coil device(from assembly 200) positioned over a graphical representation of asubject's head (e.g., head 300). It also shows that the user has enabledpower to the left arm and a “Bore Camera” (or cameras, such as camera190 shown in FIGS. 1C-3C, which may be exposed without being affected tothe magnetic field created in an MRI environment) and the digitizing arm(note that the unselected “Right Arm” and “Base Motor” buttons areunselected and grayed out).

FIG. 8 illustrates a version of CSD 401 indicating that a user hasselected to place the right arm in stereotaxy mode and Z Axis Lock mode,where the tool that has been selected for use by the right manipulatoris shown on the right lower part of the screen (and is the same biopsytool shown in FIGS. 1C-3C). The mode of the displayed manipulator shownin FIG. 8 was achieved through a user's selection of stereotaxy rightarm (as shown in the buttons in FIG. 5), master/slave mode via selectionof the Master/Slave button shown in FIG. 8, and the enablement of theright arm by selecting “Right Arm” in the “Arm Enable” box of the “ModeControls” tab shown in FIG. 8. Next, the user enables the input deviceassociated with the right manipulator by depressing button 29 on righthand controller 20. Once the input device is enabled, and because theuser has not put the system into training/simulation mode, the user canmanipulate the enabled input device to put the manipulator into theposition and orientation desired by the user for movement of the toolalong a single axis. Once the manipulator is in position, the user candisable control of the manipulator by again pushing button 29;otherwise, the user can proceed to enabling the z-axis lock for the toolheld by that manipulator by (in the depicted embodiment) selecting“Right Tool” in the “Z Axis Lock” box of the “Mode Controls” tab shownin FIG. 8. In this mode, the tool held by the manipulator will onlytravel along the axis defined (in the depicted embodiment) by the upperportion of the tool where it is held by the tool holder portions coupledto the end effector of the manipulator (which, in this embodiment, is alongitudinal axis that is centered in the entire length of tool), suchtravel occurring in the forward or backward directions depending on theuser's motion of the input device. When the user no longer desires tolock the motion of the tool to such axis, the user can push the same“Right Tool” button to disable that mode.

FIG. 9 illustrates a version of CSD 401 indicating that a user hasselected microsurgical mode and simulation mode, and enabled bothmanipulators (which are both highlighted) and selected tools for them. Auser may enable the Z Axis Lock function for the tools of both arms fromthis version of CSD 401. The selected tool for each manipulator is shownto the side of the manipulator (bipolar forceps on the left and biopsytool on the right).

FIG. 10 illustrates a version of CSD 401 in which a simulated stereotaxyleft arm mode has been selected (by, for example, selecting the“Stereotaxy Left Arm” button shown in FIG. 5), the left arm has beenenabled, the Z Axis Lock function has been selected for the left tool.

Referring now to FIG. 11, MRID 402 depicts a GUI that allows a user totoggle between 2D and 3D views taken with a 3-D imaging modality (suchas an MRI machine) of a portion (such as the head) of a subject, asreflected in the 2D tabs “2D Tools” and “2D View” at the bottom left ofthe screen and in the 3D tabs “3D Tools” and “3D View” at the bottomright of the screen. In FIG. 11, an indicator (in this example,crosshairs) is displayed of the location of the tip (e.g., tip 160) ofthe relevant tool (e.g., surgical tool 150, or, in other embodiments,the terminal end of an extension line that extends from the tool tip adistance selected using the slider bar shown underneath the “Tool TipExtension:” box beneath the tool that is being tracked) within theportion of the subject displayed in the image or dataset of images(which can be a 3D image made of multiple slices of 2D images). In theFIG. 11 version of MRID 402, the 2D Tools tab has been selected, and atwo-dimensional image is shown overlaid by the crosshairs indicatorshowing the location of the tip of the right tool within the subject.These crosshairs appear in response to a user selecting the “Track”button beneath the section for the relevant tool(s). By selecting theTrack option on MRTD 402, a user can view the MRID as he or shemanipulates the relevant input device to follow (or track) the locationof the tool tip (or tool tip extension line end point, and regardless ofwhether the user is in simulation/training mode) relative to thesubject. as it travels through the subject.

In the version of MRID 402 shown in FIG. 12, the 3D Tools tab has beenselected and the location of the tip of the tool relative to thesubject's head is shown in 3D, where the 3D image is shown in thisembodiment cut away on a plane that is normal to the axis along whichthe tool tip will travel, as a result of the selection of the “PlaneCut” button within the “Right Tool” box near the right of the screen. Auser can manipulate the orientation of the 3D image through any suitableinput device (e.g., a space ball) to move the displayed image and theoverlaid tool so as to provide a desired view of the tissue affected bythe proposed tool position and path. This overlay feature becomesavailable following the physical and MRI registration process andreceipt of tool selection. Selection of the “Simple” button will replacethe tool image with, for example, a thin red line of the same length asthe tool so as not to obstruct the view of small structures. Selectionof the “Wedge Cut” button will cut into the displayed 3D image at thelocation of the tool tip/extension line end by cutting away a wedge toreveal three orthogonal planes (e.g., sagital, axial, coronal), wherethe tip of the tool/extension line end is at the juncture of the threeplanes. These cut-away options allow a user to evaluate the internalstructure of the three-dimensional MR image to determine an optimal pathof the relevant tool during a procedure.

An exemplary embodiment of one series of steps that can be used,following the registration procedure described above, to set up andexecute a procedure (for example, an automated biopsy) is providedbelow. A user may first select a mode on the CSD, such as StereotaxyLeft Arm Mode, and then enable the left arm and power on the borecamera(s). The user may then choose the Simulation Mode on the CSD todisengage the left manipulator (which may, for example, be a leftversion of manipulator 100 from FIGS. 1C-3C or one of the manipulatorsshown in the '316 patent) from the motion of the relevant input device(such as input device 20). On the MRID, the user may then select the 2DTools tab and the Track mode/function in the “Left Tool” box, causingthe crosshairs to appear overlaying the relevant 2D image of the subjectwhen the 2D mode is selected. A user may select a non-zero “Tool TipExtension” value, using the slider bar, if a tool tip extension line isdesired. If the Tool Tip Extension function is set greater than 0.0 mm,the crosshairs will track the location of the end of the extension line.If this parameter is set at zero, the tracking function will illustratecrosshairs on the 2D slice image at the location of the tip (distal end)of the tool. As the tool or extension line passes through the subject(e.g., the brain), subsequent 2D images (e.g., 2D slices) are shown.Likewise, if the tool or extension line is withdrawn from the subject,prior 2D slices are shown.

In this exemplary embodiment, the user can grasp the left input deviceand enable virtual or simulated motion of the tool by actuating (e.g.,via use of the thumb) an enable button (e.g., button 29) on the inputdevice. The user can then take the input device, and based on visualcues gained from toggling, as desired, between the 2D and 3D MRID views,move the virtual manipulator shown on the CSD and the manipulator'ssurgical tool to the area of the intended target. In certainembodiments, the CSD and the 2D and 3D MRID images can update in realtime to show the location of the virtual (when in simulation mode)manipulator and its tool.

When the user has determined a desired target location, the user maydisable the input device so that movement of the input device does notlead to further movement of the virtual manipulator and its tool. Oneither the 2D or 3D version of the MRID screen under “Automated Biopsy”(see, e.g., FIGS. 11 and 12), a user can then push “Target” to selectthe target location for the procedure, which is stored in terms of X, Yand Z Cartesian coordinates of the tool tip in image space (e.g.,magnetic resonance imaging space), which is then transformed to robotspace. These coordinates are registered as the tool tip if the extensionline value equals zero, or as the end of the extension line if thatvalue is greater than zero; as a result, a target location indicatorwill appear at the crosshairs location (for example, a red circle) inthe 2D view and at the tool tip or extension line end location in the 3Dview denoting the intended target.

A user can then enable the input device if it has been disabled (and inthe same way that the input device was disabled) and cause the tool tipor extension line end to move toward the intended insertion point forthe subject. A path indicator (for example, a green line) can then bevisible in the 3D view that links the tip/extension line end to theselected target so that the user can see the trajectory and any tissuethat will be penetrated or disturbed by the proposed tool tip path. Theuser may then move the input device to the desired entry point (whichcould be, for example, at the surface of the brain or head, or a smalldistance outside the head). If a burr hole has already been made, a usermay ensure that the path goes through the burr hole without contactingthe head. The user may then, but need not, disable the input device whenthe entry point and trajectory are acceptable.

A user may then push the button labeled “Start Point” on either the 2Dor 3D version of the MRID and an indicator (e.g., a green circle) willappear at the crosshairs location in the 2D view and at the tool tip orextension line end location in the 3D view denoting the intended startpoint, which is stored in terms of X, Y and Z Cartesian coordinates ofthe tool tip in image space (e.g., magnetic resonance imaging space),which is then transformed to robot space. These coordinates areregistered as the tool tip if the extension line value equals zero, oras the end of the extension line if that value is greater than zero. Inthis embodiment, the indicator will change in some way (e.g., the greenline will turn red) to denote that the line (or path) is set, and thestart point, termination point and trajectory (or path) will appear onthe CSD. If a user desires to change the location of the Start Point orthe Target, the user can use the input device to move the simulated tooltip/extension line to a new location and push the “Start Point” or“Target” button again in either the 2D or 3D version of the MRID. Incertain embodiments, the system will ask the user to push the relevantbutton a second time to confirm replacement of the old point with a newpoint. After an acceptable trajectory is chosen, the user can exit thesimulation mode on the CSD by designating that button again (e.g., bytouching it on the screen again).

After selecting/determining the desired trajectory for the chosen tool,a user can execute the automated move by choosing the master/slave modeon the CSD, enabling the input device (e.g., by depressing button 29),and moving the input device to cause the manipulator (while watching theMRID and/or the bore camera or field camera image shown on displayscreen 403 shown in FIG. 4) to move to a location close to the startpoint selected for the movement and to be in an orientation that is asclose to the selected trajectory as possible. The user may then disablethe input device.

On the MRID, under “Automated Biopsy” in either the 2D or 3D view, theuser can push the “Tool Align” button (see, e.g., FIGS. 11 and 12) andthe manipulator will move to align with the programmed trajectory andplace the tool tip at or near the selected start point (such asapproximately two centimeters radially outward from the start pointalong the programmed trajectory). The user may then push the “Execute”button (under “Automated Biopsy”), and the user may be prompted toenable the input device to begin the automated movement (e.g., anautomated biopsy).

The user may then grasp the input device and enable the system to beginthe automated biopsy by enabling the input device in the same way it hasbeen previously enabled (e.g., by pushing button 29). Taking this stepcauses the user to hold the input device in order for the procedure totake place. As a result of enabling the input device, the tool may moveforward at a predetermined rate (which can be set in an initializationfile) to the target location, at which point the surgical tool canperform a pre-programmed function, such as removing biopsy material. Incertain embodiments in which the surgical tool is a biopsy tool equippedwith two small sharpened scoops that open away from each other aboutaxes that are normal to the longitudinal axis of the tool, the surgicaltool's scoops will open, rotate 90 degrees clockwise, and close again,capturing tissue as a result. The surgical tool can then reversedirection straight out along the insertion trajectory.

If a problem is encountered during execution of the automated move, theuser can disable the input device (e.g., by depressing button 29) tostop the move. The CSD will then present the user with a selection box,such as the one shown in FIG. 13, that includes options to stop,continue, and reverse direction. Once a selection is chosen, the toolwill move again when the user enables the input device.

In addition to providing a single axis lock for movement of a givensurgical tool during any procedure, embodiments of the present methods,devices, and systems may therefore also allow a user (e.g., a surgeon)to simulate multiple paths for a surgical tool prior to conducting theactual surgical procedure, evaluate those paths for the tissue they mayaffect, and choose a desired path by selecting a target point and astart point. The present devices and systems are configured to limit(electronically) the tool to a linear path; as a result, only a startpoint and a target point are needed to determine the tool path.Embodiments of the present devices and system may also comprise multiplesafety features to allow the user to maintain control of the tool.

Embodiments of the present methods may be coded as software stored onany suitable computer readable media (e.g., tangible computer readablemedia), such as any suitable form of memory or data storage device,including but not limited to hard drive media, optical media, RAM, SRAM,DRAM, SDRAM, ROM, EPROM, EEPROM, tape media, cartridge media, flashmemory, memory stick, and/or the like. Tangible computer readable mediaincludes any physical medium that can store or transfer information.Such embodiments may be characterized as tangible computer readablemedia having (or encoded with) computer executable (e.g., machinereadable) instructions for performing certain step(s). The term“tangible computer readable medium” does not include wirelesstransmission media, such as carrier waves. The term “computer readablemedium,” however, does cover wireless transmission media, and someembodiments of the present methods may include wireless transmissionmedia carrying the computer readable instructions described above. Thesoftware can be written according to any technique known in the art. Forinstance, the software may be written in any one or more computerlanguages (e.g., ASSEMBLY, PASCAL, FORTRAN, BASIC, C, C++, C#, JAVA,Perl, Python) or using scientific packages like, but not limited to,Matlab®, R, S-plus®, and SAS®. The code may be to enable it to becompiled on all common platforms (e.g., Microsoft®, Linux®, AppleMacintosh® OS X, Unix®). Further, well-established cross-platformlibraries such as OpenGL® may be utilized to execute embodiments of thepresent methods, devices and systems. Multi-threading may be usedwherever applicable to reduce computing time on modern single- andmulti-processor based hardware platforms. As discussed above andillustrated in the figures, the software may include a GUI, which mayprovide a user with a more intuitive feel when running the software.Different fields may be accessible by screen touching, a mouse and/orkeyboard. Alarms, cues, and the like may be done via pop-up windows,audible alerts, or any other techniques known in the art.

Some (up to all) of the steps described in the sections above may beimplemented using a computer having a processor (e.g., one or moreintegrated circuits) programmed with firmware and/or running software.Some (up to all) of the steps described in the sections above may beimplemented using a distributed computing environment, which is oneexample of a computer system. In a distributed computing environment,multiple computers may be used, such as those connected by any suitablenumber of connection mediums (e.g., a local area network (LAN), a widearea network (WAN), or other computer networks, including but notlimited to Ethernets, enterprise-wide computer networks, intranets andthe Internet, and the connections between computers can be wired orwireless). Servers and user terminals can be part of a given computersystem. Furthermore, embodiments of suitable computer systems may beimplemented on application specific integrated circuits (ASICs) or verylarge scale integrated (VLSI) circuits, and further (or alternatively)may be configured to use virtualization of resources, virtual computing,and/or cloud computing to achieve the specified functions. In fact,persons of ordinary skill in the art may utilize any number of suitablestructures capable of executing logical operations in order to achievethe functions described above in a computer system consistent with thisdisclosure.

Descriptions of well known processing techniques, components andequipment have been omitted so as not to unnecessarily obscure thepresent methods, devices and systems in unnecessary detail. Thedescriptions of the present methods, devices and systems are exemplaryand non-limiting. Certain substitutions, modifications, additions and/orreanangements falling within the scope of the claims, but not explicitlylisted in this disclosure, may become apparent to those of ordinaryskill in the art based on this disclosure. For example, while one MRIDis disclosed that allows a user to toggle between the display of 2D and3D images, in alternative embodiments two separate display screens maybe used for 2D and 3D images, respectively. As another example, while anautomated movement for a biopsy of brain tissue has been described aboveas an example of a suitable movement that can be pre-programmedaccording to the techniques disclosed above, there are many othersurgical and/or diagnostic movements that can be automated using thepresent techniques, including breast biopsies, the implantation ofdrugs, the implantation of electrodes (e.g., for epilepsy), theimplantation of stem cells, and the drilling of bone spurs fromvertebrae without line of sight, among others. Furthermore, it will beappreciated that in the development of a working embodiment, numerousimplementation-specific decisions must be made to achieve thedevelopers' specific goals, such as compliance with system-related andbusiness-related constraints, which will vary from one implementation toanother. While such a development effort might be complex andtime-consuming, it would nonetheless be a routine undertaking for thoseof ordinary skill in the art having the benefit of this disclosure.

The appended claims are not to be interpreted as includingmeans-plus-function limitations, unless such a limitation is explicitlyrecited in a given claim using the phrase(s) “means for” and/or “stepfor,” respectively.

1-14. (canceled)
 15. A computer system configured to perform at leastthe following functions: receive a command designating a target locationfor a tool operatively associated with a robotic arm of a medical robot,the robotic arm being configured for use in surgery; receive a commanddesignating a second location for the tool; receive a command to beginan automated movement of the tool along a single axis defined by thesecond and target locations; and send one or more commands sufficient tocause the robotic arm to effect the automated movement, thereby movingthe surgical tool from the second location to the target location, theautomated movement occurring at a pre-determined rate which is distinctfrom a real-time response to manipulation of an input device.
 16. Thecomputer system of claim 15, further configured to perform at least thefollowing function: display a simulated representation of the surgicaltool that includes an indicator showing a path from the second locationto the target location.
 17. The computer system of claim 15, wherein thecomputer system is useful in simulating, planning and/or executing anautomated surgical procedure, wherein the medical robot has two arms,and wherein the computer system is further configured to perform atleast the following functions: display an image of one or both arms ofthe medical robot; receive a command to operate in a simulation mode;display a two-dimensional (2D) image of a portion of a subject; displaya target location indicator overlaid on the 2D image; move the targetlocation indicator in response to input from an input device; display animage representative of a tool overlaid on a three-dimensionalrepresentation of a portion of the subject, the tool having a tool tipor a line extending from the tool that is shown in the same relativelocation as the target location indicator; move the tool tip or the linein response to user input from the input device; manipulate theorientation of the 3D representation in response to input from the inputdevice; receive a command designating a target location for the tool;receive a command designating the second location for the tool fromwhich the tool will move toward the target location during an automatedmovement; and move the medical robot in response to a user command tobegin the automated movement such that the tool moves along the singleaxis defined by the second and target locations.
 18. (canceled)
 19. Thecomputer system of claim 17, wherein the tool that moves along thesingle axis is operatively associated with one of the arms of themedical robot, and the move occurs only if data is received indicatingan input device linked in a master-slave relationship to that arm. 20.The computer system of claim 19, further configured to perform at leastthe following additional function: cause manipulation of the tool at thetarget location.
 21. A computer readable medium comprising machinereadable instructions for performing at least the functions of claim 15.22. The computer system of claim 17, further configured to perform atleast the following function: display a different 2D image of a portionof the subject in response to user input;
 23. The computer system ofclaim 17, further configured to perform at least the following function:receive a command to stop the automated movement before the automatedmovement is complete.